Amendments to the Title: 

Please delete the title "TEST RESOLVER MANAGEMENT SYSTEM" and replace it with 
the following replacement title: 



DYNAMIC ON-LINE SCORING GUIDE AND METHOD 
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Amendments to the Specification: 

At the top of page 1 of the specification, insert the following paragraph: 

This is a continuation of application of Serial No. 10/425,775, filed on April 29, 2003, 
which is a continuation of application Serial No. 09/660,204, filed September 12, 2000, now U.S. 
Patent No. 6,558,166 Bl, which is a continuation of application Serial No. 09/141,804, filed on 
August 28, 1998, now U.S. Patent No. 6,168,440 Bl, which is a continuation of application 
Serial No. 09/003,979, filed on January 7, 1998, now abandoned, which is a continuation of 
application Serial No. 08/561,081, filed November 20, 1995, now U.S. Patent No. 5,735,694, 
which is a continuation of appHcation Serial No., 08/290,014, filed August 12, 1994, now U.S. 
Patent No. 5,558,521, which is a division of application Serial No., 08/014,176, filed February 5, 
1993, now U.S. Patent 5,437,554. U.S. Patent No. 5,690,497, U.S. Patent No. 5,458,493, U.S. 
Patent No. 6,155,839, U.S. Patent No. 6,183,261, U.S. Patent No. 6,168,440, U.S. Patent No. 
5,735,694, U.S. Patent No. 5,466,159, U.S. Patent No. 5,437,554, and U.S. Patent No. 6,558,166 
Bl are hereby incorporated by reference in their entirety. 

Please delete the paragraph beginning at page 3, line 15, and insert the following 
paragraph: 

The present data item resolver management system facilitates consistent, accurate, and 
high quality scoring of data items. The system and method provide an on-line scoring guide to 
dynamically provide a resolver with scoring goals for a data item displayed to the resolver. The 
method and system further store rules which are related to procedures for resolving data items 
questions. When data items are presented to a resolver for scoring, the corresponding rules for 
scoring the displayed data item are presented. 

Please delete the paragraph beginning at page 3, line 21. 

Please delete the paragraph beginning at page 4, line 3. 
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Please delete the paragraph beginning at page 4, line 15. 

Please delete the paragraph beginning at page 5, line 12, and insert the following 
paragraphs: 

Fig. 12A is a flow chart of resolver monitoring and feedback. 

FIG. 12B is a flow chart of retrieval of information for resolver monitoring based on the 
type of performance feedback. 

Please amend the paragraph beginning at page 17, line 13 as shown below: 

FIGS. 9 A and 9B are Fig. 9 is a flow chart showing additional typical functions of the 
collaborative scoring. At steps 92 and 93, the system displays the items to resolvers 1 and 2 for 
scoring. The system may further track the average scores of resolvers and not send the same test 
item to two resolvers who have provided average scores within a predefined range. This also 
helps to achieve consistency in scoring. For example, if two scorers each have provided high 
average scores in the past, as determined by the system, these two scorers should preferably not 
be collaboratively scoring the same test items, since it could result in "inflated" scores for 
particular test items. 

Please insert the following paragraph at page 20, line 22: 

Figure 12A is a flow chart of typical resolver monitoring and feedback. The 
primary factors in monitoring performance typically include: (1) validity; (2) reliability; 
and (3) speed. In monitoring these factors, the system promotes repeatability of scoring. 

These factors may be monitored by tracking a resolver's performance against past 
performance of the resolver or against some known goal. 

On page 22, line 6, insert the following paragraph: 

FIG. 12B is a more detailed flow chart of step 121 from FIG. 12A. The flow chart of Fig. 
12B shows the retrieval of information for resolver monitoring based on the type of performance 
feedback. At step 140, the system determines the type of performance feedback for monitoring 
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the resolver, such as vahdity, rehabiUty, or speed. At steps 141, the system retrieves information 
for providing performance feedback related to validity, which may include receiving first 
performance levels corresponding to a resolver's scoring of test answers at predetermined past 
intervals in time, and receiving a second predetermined performance level. At steps 142, the 
system retrieves information for providing performance feedback related to reliability, which 
may include receiving a first score corresponding to a first resolver's scoring of a test item during 
a first time period, and receiving a second score corresponding to the first resolver's scoring of 
the test item during a second time period. At steps 143, the system retrieves information for 
providing performance feedback related to speed, which may include receiving a first scoring rate 
corresponding to a first resolver's scoring of test answers, including scoring test answers at 
predetermined past intervals in time; receiving an average scoring rate corresponding to a 
selected group of resolvers' scoring of the test answers; receiving a second scoring rate 
corresponding to a second resolver's scoring of the test answers; and receiving a second scoring 
rate corresponding to the first resolver's scoring of the test answers, including scoring of the test 
answers at a present time. 
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Amendments to the Drawings: 



The drawings include changes to Figs. 5, 8, 9 and 12. The set of formal drawings replace the 
original sheets. The drawings have been changed as follows: 

Figure 5 has been amended to add flow direction indicators; 
Figure 8 has been amended to delete sections 141 and 142 and add section 140; 
Figure 9 has been changed to separate Fig. 9 into new Figs. 9 A and Figure 9B; 
Figure 12 has been changed to separate Fig. 12 into new Figs. 12A and Figure 12B. Fig. 
12B is a more detailed flow chart of Fig. 12A showing the monitoring of performance feedback. 

A submission of 16 sheets of Formal Drawings are being filed currently herewith. 
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